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(57) Abstract: The invention provides a system for enhancing 
the ability of a surgeon to access a target site within a patient. The 
system includes a data file containing volumetric scan data of a 
region of the patient that includes the target site, a display device, 
and a movable imaging tool for producing on the display device, 
an image of visible patient structure seen by the tool. A com- 
puter in the system operates to (i) determine the position and/or 
orientation of the tool in the frame of reference of the patient, 
(ii) identify the scan-data coordinates (either x, y or x, y, 7. co- 
ordinates) of the target site, and (iii) project on the video image 
on the display device, indicia that indicate the lateral position of 
the target site with respect to the patient structure imaged on the 
display device. 
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ENDOSCOPIC TARGETING METHOD AND SYSTEM 

Field of the Invention 
5 This invention relates to image-guided surgery, and in particular, to an 

endoscopic targeting method and system. 

Background of the Invention 

Endoscopic surgical tools are used in a variety of surgical procedures. 

10 Typically, such tools include an optical system for visualizing patient structure 
at or near a target site, and a surgical tool for carrying out desired operations 
at the site, e.g., removal of tissue for biopsy, surgical removal of necrotic or 
tumorous tissue, surgical repair of tissue structure, etc. U.S. Patent Nos. 
5,928,137, 5,968,061, 5,681,262, 5,840,017, 5,840,014 and 5,830,126 

1 5 describe representative types of endoscopic surgical tools. 

In an endoscope-guided operation, the surgeon will need to know in which 
direction, and what distance to advance the tool in order to optimally access 
the target site. Since an endoscopic tool can only view surface structure, the 
surgeon will often have difficulty in locating and/or accessing a target site, 

20 which is likely to be hidden from endoscopic view. 

It would therefore be desirable to provide an endoscopic targeting 
method and system to assist a surgeon in performing an endoscopic surgical 
procedure or endoscopic examination of a patient. 

25 Summary of the Invention 

The invention includes, in one aspect, a system for enhancing the 
ability of a surgeon to access a target site within a patient. The system 
includes a data file containing volumetric scan data of a region of the patient 
that includes the target site, a display device, a movable imagining tool for 

30 producing on the display device, an image of visible patient structure seen by 
the tool, where the position of the tool is tracked relative to the position of the 
patient, and a computer operatively connected to data file, display screen, and 
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tracking device. 

The computer operates to (i) determine the position and/or orientation of 
the tool in the frame of reference of the patient, (ii) identify the scan-data 
coordinates (either x,y or x,y,z coordinates) of the target site, and (iii) project on 
5 the video image on the display device, indicia that indicate the lateral position of 
the target site with respect to the patient structure imaged on the display 
device. 

In one embodiment, in which the indicia give 2-D information about the 
target site, the projected indicia define an area on the image that borders the 

10 lateral position of the target site. The area is to defined by a function related to 
the depth of the depth of the target site below the patient structure imaged on 
the display device, such as an error function related to the degree of uncertainty 
in patient target site in the imaged structure. 

The computer may further function to determine whether the target site 

15 is located within the lateral field of the displayed patient-structure image, and if 
it is not, to project on the displayed image, indicia indicating the lateral direction 
of the target site with respect to the imaged patient structure. 

The system may further include a calibration tool operatively connected 
to the computer for supplying to the computer, information about the imaging 

20 tool's field of view, position, orientation and direction. 

Where the imaging tool is an endoscopic surgical tool having an interior 
channel through which a surgical tool can be manipulated, the system may 
further include structure for tracking the position of said surgical tool relative to 
the imaging element in the imaging tool, and the computer may further function 

25 to project on the video image on the display device, indicia that provide 
information about the position of the target site in relation to the end of the 
surgical tool. 

In another aspect, the invention includes a method for enhancing the 

ability of a surgeon to access a target site within a patient. The method 

30 includes the steps of storing in a data file, volumetric scan data of a region of 

the patient, including a target site within the patient region, moving an imaging 

tool for producing on a display device, an image of visible patient structure seen 
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by the tool, and tracking the position of the tool relative to the position of the 
patient. The method employs a computer screen, for (i) determining the 
position and/or orientation of the tool in the frame of reference of the patient, (ii) 
identifying the scan-data coordinates (either x,y or x,y,z coordinates) of the 
5 target site, and (iv) projecting on the video image on the display device, indicia 
that indicate the lateral position of the target site with respect to the patient 
structure imaged on the display device. 

Various above-mentioned embodiments applicable to the system of the 
invention are also applicable to the method. 
10 These and other objects and features of the invention will be more fully 
appreciated when the following detailed description of the invention is read in 
conjunction with the accompanying drawings. 

Brief Description of the Drawings 
15 Fig. 1 illustrates components of the system of the invention, shown in a 

surgical setting; 

Fig. 2 is a flow diagram of the operation of the system of the invention; 
Fig. 3 is a flow diagram of the system operation in generating 2-D and 3- 
D indicia; 

20 Fig. 4 shows a video image display where the target site is out of the 

field of the image; and 

Figs. 5A-5C illustrate various states of 2-D indicia projected on an 
endoscopic image to assist a surgeon in accessing a target site in an 
endoscopic surgical procedure. 

25 

Detailed Description of the Invention 

Fig. 1 illustrates components of a system 10 of the invention, shown in 

operative relationship to the head region of a patient 12 supported on a 

surgical table 14. For purposes of illustrating the describing the invention, the 

30 patient region of interest-the surgical target region-is within the a region of 

the patient's body head accessible by an endoscopic tool inserted through the 

throat or nostrils, e.g., a region of the brain near a sinus cavity. It will be 
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appreciated that the invention is applicable to any body region, e.g., body 
joints, where surgery with an imaging tool, e.g., an endoscope, is carried out. 
For use in tracking patient position during surgery, a plurality of tracking 
elements, such as LEDs 17, are attached at fixed positions on the patient 
5 surface. 

The system includes an endoscopic surgical tool 16, or other surgical 
imaging tool, such as a microscope ocular, or simple endoscope. Tool 16 
includes an elongate tool body 18 which terminates at its distal end in an 
endoscopic lens 20, and which provides an interior channel through which a 

10 surgical appliance 22 may be moved. Exemplary endoscopic tools are those 
described in U.S. Patent Nos. U.S. Patent Nos. 5,928,137, 5,968,061, 
5,681,262, 5,840,017, 5,840,014 and. 5,830,1 26. In addition, tool 16 includes 
a plurality of tracking elements, such as LED's 24, used in tracking the 
position, direction and orientation of the tool. 

1 5 In the embodiment of the system shown, a sensing unit is part of a 

position tracking system, which is preferably an optical tracking system 
(hereafter "OTS") having sensing unit or sensor 26 mounted overhead in view 
of the operating table scene, and tracking elements, e.g., LEDs attached to 
the patient and endoscopic tool as noted above. The LED's attached to the 

20 movable objects are preferably designed to emit streams of pulsed infrared 
signals which are sensed by a plurality of infrared detectors (not shown) 
contained in the sensing unit. The sensing unit and movable objects are 
connected to the computer, which controls the timing and synchronization of 
the pulse emissions by the LED's and the recording and processing of the 

25 infrared signals received by the detectors. 

The OTS further includes software for processing these signals to 
generate data indicating the position and orientation (direction and twist) of 
the movable objects in the frame of reference of surgical station. The OTS 
may generate the position data on a real-time basis, so that as the 

30 endoscopic tool is moved, for example, or the patient moves, their positions in 

a common coordinate system, e.g., the patient's coordinate system, are 

known. One preferred OTS is described in PCT application WO 9900052 A1 
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for Image Generation of Three Dimensional Object, which is incorporated 
herein by reference. 

A variety of alternative tracking devices and methods for tracking the 
position of a movable tool with respect to a patient, such as those detailed in 
5 U.S. Patent Nos. 5,198,877, 5,987,349, and 5,622,170 are also available. 

A calibration tool 28 in the system is used in calibrating the image seen 
by the endoscope lens with a known endoscope field of view, instrument 
position (e.g., x,y,z coordinates of a point in the endoscope lens) and 
orientation, which includes both instrument direction (the angle of the tool with 

10 respect to a fixed position) and angular orientation or twist (the angular 
position of the tool about its long axis). To this end, tool 28 has an elongate 
cavity 30 which receives the lens-end of the endoscopic instrument, and a 
multi-point pattern (not shown) at the lower end of the barrel which is imaged 
by the instrument in its calibration position. The calibration tool is also 

15 equipped with tracking elements, such as LEDs 32, used by the OTS to 
determine the position of the calibration tool. 

With the endoscopic instrument placed in the calibration tool, the 
pattern image seen by the instrument is recorded, as is (i) the position and 
orientation of the calibration tool, and (ii) the position and orientation of the 

20 endoscopic instrument in the calibration tool. This endoscope calibration 
information is used, in part, in registering the patient-surface image seen by 
the endoscope lens with the patient-structure coordinates of the pre-operative 
scan data. The calibration tool is described in U.S. patent application Serial 
No. 60/193,209, filed March 30, 2000 for "Device for Calibrating a Medical 

25 Instrument, Including Determining Field-of-View and compensation for Lens 
Distortion and Offset", which application is hereby incorporated by reference. 

System 10 includes a data-scan file 38 which stores pre-op scan data, 
typically obtained at a scanning station. The scan-data consists of voxel 
coordinates, corresponding scan-data, e.g., density values, for each voxel, 

30 and coordinates for registration features, e.g., LEDs 17 or other fiducials, 

which are present on the patient during scanning, and are maintained in place 

during the subsequent surgical operation. The scan data file is operatively 
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connected to computer 36, which carries out the data processing operations 
of the system, to be described. Also connected to the computer are sensor 
26 and a display device 40, such as a conventional color monitor screen. 

As will be described in greater detail below, the computer is operatively 
5 connected to data file, display screen, and tracking device, for (i) determining 
the position and/or orientation of the instrument in a patient coordinate system, 
(ii) identifying the scan-data coordinates of the target site in the patient 
coordinate system, and (iii) projecting on the video image on the display device, 
indicia that indicate the lateral position of the target site with respect to the 

10 patient structure imaged on the display device. 

Looking now at Fig. 2, volumetric scan data of the patient region of 
interest is stored in scan-data file 38. Also prior to the surgical procedure, the 
endoscopic tool is calibrated, as above, and information relating to the pattern 
image seen by the endoscope in relation to endoscope position, direction, and 

15 orientation is stored in an endoscope calibration data file 42. By tracking the 
position, orientation, and direction of the instrument in space, as at 54, and 
tracking the position of the instrument with reference and knowing the position 
of the patient in space (both determined from the OTS), the position, direction, 
and orientation of the instrument can be placed in the patient frame of 

20 reference, i.e., coordinate system, as at 56. Alternatively, the position of the 
instrument can be tracked directly in the patient frame of reference to provide 
the same information. At the same time, knowing the endoscope field of view, 
the view seen by the endoscope within the body can be matched with the 
stored CT data, which is also in the frame of reference of the patient. This 

25 matching is effective to superimpose the endoscopic image on a surface image 
of patient structure which would be reconstructed from the patient CT data. 
The endoscopic image is displayed conventionally on display device 40, as 
shown at 58. 

Prior to the surgery or during surgery, the surgeon has identified the 

30 target site of interest, and the coordinates of this site in the scan data are 

specified, e.g., by clicking on target structure revealed in a perspective view of 

the target region reconstructed from the CT data. The target-site coordinates 
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specified are stored as at 44. 

For purposes of explanation, it is convenient to consider the patient 
coordinate system in terms of a z axis extending between a selected point on 
the endoscopic tool, e.g., a point at the center of the lens, and a target site 
5 point. The x,y plane is then a plane normal to this axis and containing (having 
x,y coordinates in common with) the video image seen by the endoscopic tool. 
The x,y distances within the video image are known from the previously-input 
field of view of the lens. If the x,y coordinates of the target site are outside of 
the image, as tested at 46, the computer will construct an indicium, e.g., arrow, 

10 that is placed at the periphery of the video image to indicate the x,y direction of 
the target site in relation to the image on the display screen. This feature is 
illustrated in Fig. 4, which is intended to represent a display screen showing a 
video image 76 of patient structure and an arrow 78 indicating that the target 
site is "off" the screen in the direction of the arrow. This allows the surgeon to 

1 5 move toward the target site by moving in the direction of the arrow. 

If the target site x,y coordinates are within the video image x,y 
coordinates, the system then determines the distance along the z axis between 
the instrument and target site, namely (x 2 + y 2 + z 2 ) 1/2 , and may display this 
distance to the user at the display screen, as at 60. The system also 

20 determines indicia which will be placed on the video image to indicate to the 
user the following types of information (i) the x,y (lateral) position of the target 
site, (ii) the z distance from the instrument to the target site, and optionally, (iii) 
the uncertainty in target-site position. 

To illustrate, and with reference to Fig. 3, the system-operation steps at 

25 the left in the figure operates first to calculate a target-site position error, as at 
64. This error represents the uncertainty in determined target-site position due 
to uncertainty in patient position, instrument position, and uncertainty about the 
target site within the patient. This error is a function of distance, since the 
degree of position uncertainty increases with distance from the target site, and 

30 may thought of as a conical function emanating from a small error "circle" at the 

target and expanding on moving away from the target site, in this case, along 

the z axis. The conical slices along the z axis then define an area (circle) 
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whose size (diameter) is related to the error function. This calculation is made 
at 66 in the figure. 

The indicia which define the area in the 2-D display are positioned so 
that the center of the area, e.g., circle, has the x,y coordinates of the target site. 
5 The size of the defined area is then adjusted, as at 66, to correspond (by same 
scaling factor) relate to the z-axis distance to the target. The thus-determined 
indicia are then displayed on the video image, as at 68. 

Figs. 5A-5C show three hypothetical endoscopic video image 80a, 80b, 
80c having 2-D indicia placed on the image in accordance with the invention. 
10 The indicia in this example are four arrows, such as arrows 82, that define a 
circular area (within the dotted line 84). The x,y coordinates of the indicia (the 
center of the circle) correspond to the x,y coordinates of the target site and the 
diameter of the circle, to the z-axis distance between the instrument and target 
site. 

15 The three figures show the change in indicia as the instrument is moved 

from a position down and to the left of the target site (Fig. 5A) progressively 
closer in x,y,z coordinates to the target site. (Figs. 5B and 5C). When the 
instrument is moved to the target site (Fig. 5B), the arrows should be at the 
center of the screen, with the diameter of the circle then representing the 

20 uncertainty in target-site position. 

The method and system have been described above with reference to 
tracking the "view" point of the endoscopic image (the lens elements) toward a 
target site. It will be appreciated that the same principles and operations can 
be adapted to tracking the surgical tip of the instrument toward the target site. 

25 In this case, a second tracking structure which tracks the z-axis movement of 
the instrument tip with respect to the endoscope lens is used by the computer 
to determine the position of the tip along the z axis. The indicia displayed on 
the video image are adjusted accordingly to show the shorter distance between 
the tip and target site. 

30 Alternatively, one set of indicia, e.g., indicia of one color, may be used to 

indicate the position of the instrument lens element from the target, and a 

second set of indicia having, for example, another color would indicate the 
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distance of the instrument tip from the target site. The two sets of indicia would 
allow the surgeon to move the entire instrument, and the surgical tip of the 
instrument independently to advance the tip toward the target site in an efficient 
manner. 

5 In another related embodiment, the system uses the scan data to 

generate a subsurface image, preferably a perspective subsurface image of the 
target region, as described, for example, in above noted PCT application WO 
9900052 A1 for Image Generation of Three Dimensional Object. The 
reconstructed image is displayed to the user on a display screen, e.g., 

10 alongside the endoscopic image. In accordance with the present invention, the. 
target-site indicia determined as above are then superimposed on the 
reconstructed image, employing the coordinates to the image to place and 
scale the indicia. In this embodiment, the surgeon may visualize target position 
and distance in both endoscopic and virtual images reconstructed from the 

15 scan data. 

Although the system and method of the invention have been described 
with reference to particular embodiments and implementations, it will be 
appreciated that a variety of changes and modifications can be made without 
departing from the claimed invention. In particular, a variety of 2-D and 3-D 
20 indicia and indicia formats can be constructed to assist the surgeon in 
accessing the target site. 
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WHAT IS CLAIMED IS: 

1 . A system for enhancing the ability of a surgeon to access a target site 
within a patient, comprising 

5 a data file containing volumetric scan data of a region of the patient that 

includes the target site, 
a display device, 

a movable imagining tool for producing on the display device, an image 
of visible structure seen by the tool, where the position and/or orientation of the 
10 tool is tracked with respect to the patient coordinate system, and 

a computer operatively connected to data file and display screen for (i) 
determining the position and/or orientation of the tool in a patient coordinate 
system, (ii) identifying the scan-data coordinates of the target site in the patient 
coordinate system, and (iii) projecting on the video image on the display device, 
15 indicia that indicate the lateral position of the target site with respect to the 
patient structure imaged on the display device. 

2. The system of claim 2, wherein the system includes a tracking device 
for tracking the position and direction of the instrument with respect to patient 

20 position. 

3. The system of claim 2, wherein said indicia define an area whose 
center corresponds to the lateral position of the target site, and whose size is 
related to the distance between the instrument and the target site. 

25 

4. The system of claim 3, wherein the size of the area defined by the 
indicia is calculated by an error function representing the degree of uncertainty 
in patient target site in the imaged structure. 

30 5. The system of claim 1 , wherein said computer is operatively 

connected to a calibration device for receiving information therefrom with 

respect to the imaging tool's field of view, position, direction orientation. 
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6. The system of claim 1 , wherein the computer operates to determine 
whether the target site is located within the lateral field of the displayed patient- 
structure image, and if it is not, to project on the displayed image, indicia 

5 indicating the lateral direction of the target site with respect to the imaged 
patient structure. 

7. The system of claim 1 , wherein (i) said imaging tool is an endoscope 
having an interior channel through which a surgical tool can be manipulated, (ii) 

10 the system further includes structure for tracking the position of said surgical 
tool relative to said imaging device, and (iii) the computer operates to project on 
the video image on the display device, indicia relating the position of the target 
site in relation to the tip of said tool. 

15 8. A method for enhancing the ability of a surgeon to access a target 

site within a patient, comprising 

storing in a data file, volumetric scan data of a region of the patient, 
including a target site within the patient region, 

moving an imagining tool for producing on a display device, an image of 
20 visible patient structure seen by the tool, 

tracking the position of the tool relative to the position of the patient, and 
by means of a computer a computer operatively connected to data file 
and display screen, (i) determining the position and/or orientation of the tool in 
the frame of reference of the patient, (ii) identifying the scan-data coordinates of 
25 the target site, and (iii) projecting on the video image on the display device, 
indicia that indicate the lateral position of the target site with respect to the 
patient structure imaged on the display device. 

9. The method of claim 8, wherein said tracking is done by a tracking 
30 device for tracking the position and direction of the instrument with respect to 
patient position. 
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10. The method of claim 8, wherein said wherein said projecting 
includes projecting on the video image on the display device, indicia that define 
an area whose center corresponds to the lateral position of the target site, and 
whose size is related to the distance between the instrument and the target site. 

5 

1 1 . The method of claim 10, wherein the size of the area defined by the 
indicia is calculated by an error function representing the degree of uncertainty 
in patient target site in the imaged structure. 

10 12. The method of claim 8, wherein said determining includes 

calibrating the imaging tool with respect to field of view, position, direction and 
orientation. 

13. The method of claim 8, which further includes, by means of the 
1 5 computer, determining whether the target site is located within the lateral field 
of the displayed patient-structure image, and if it is not, projecting on the 
displayed image, indicia indicating the lateral direction of the target site with 
respect to the imaged patient structure. 

20 14. The method of claim 8, wherein the imaging tool that is moved is an 

endoscope having an interior channel through which a surgical tool can be 
manipulated, and said projecting includes projecting on the video image on the 
display device, indicia relating the position of the target site in relation to the tip 
of said tool. 
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